Virtual Reality/Augmented Reality Medical Simulation
Medical imaging has become more and more used for both diagnostic/examination and therapeutic purposes in a number of medical applications, such as endoscopy for surgery, or ultrasound imaging for various gynecology and/or obstetrics applications, for instance in the embryo transfer procedure for In Vitro Fertilization (IVF). These new techniques may require dedicated training for physicians and surgeons to master the indirect hand-eye coordination required by the imaging system as well as the manipulation of the imaging tools in addition to the conventional medical instruments and procedures for a diversity of patient anatomies as may be encountered in medical practice. Computerized medical procedure training simulators may enable the physicians and trainees to develop and improve their practice in a virtual reality environment before actually practicing in the operation room.
Advanced medical procedure simulators may be based on a virtual reality (“VR”) and/or a mixed or augmented reality (“AR”) simulation apparatus by which the physician can experiment a medical procedure scenario. The VR/AR system may compute and display a visual VR/AR model of anatomical structures in accordance with physician gestures and actions to provide various feedback, such as visual feedback. In a VR system, an entire image may be simulated for display to a user, and in an AR system, a simulated image may be overlaid or otherwise incorporated with an actual image for display to a user. Various patient models with different pathologies can be selected. Therefore, natural variations as encountered over the years by practicing doctors can be simulated for a user over a compressed period of time for training purposes. The medical simulation procedure can be recorded and rehearsed for evaluation purpose. The VR/AR simulation system can also compute and provide various metrics and statistics.
VR/AR simulation systems such as the one described in U.S. Pat. No. 8,992,230 include a human anatomy model of a joint of organ in real size. The VR/AR simulation system may further comprise a medical instrument to more realistically simulate the medical procedure. The model is further adapted with sensors for tracking the position and/or orientation of both the anatomy model and the medical instrument. As described in U.S. Pat. No. 8,992,230, calibration units may be further used to automatically setup and align the VR/AR simulation system to a diversity of anatomy models and medical procedure training scenarios without requiring a cumbersome, manual calibration procedure each time a new model is adapted to the system.
A passive feedback VR/AR simulation system such as for instance the one described in U.S. Pat. No. 8,992,230 may also be used with a diversity of medical procedure training scenarios, some of which may possibly result in a mismatch between an anatomy model surface as touched by the trainee and a virtual environment surface as computed by the VR/AR simulation system and rendered on the screen. In order to further improve the passive haptic experience and increase the realism in such medical training scenarios, the VR/AR simulation system may be further adapted with space warping methods and systems as described in US patent application US20140071165.
Ultrasound Imaging Simulation
Most prior art ultrasound simulation solutions have been developed based on interpolative ultrasound simulation, as used for instance by a number of commercial ultrasound training simulators such as Medsim (http://www.medsim.com/), EchoCom (http://www.echocom.de) and MedCom/Sonofit (http://www.sonofit.de). These prior art simulators use a set of collected 2D images for each case, which are reconstructed into 3D volumes per case in an offline “case generation” stage with minor differences between individual methods in how the 3D volumes and their corresponding graphical models are generated. For instance, Dror Aiger and Daniel Cohen-Or described the use of deformable registration techniques for generating large 3D volumes from smaller swept scans in “Real-time ultrasound imaging simulation”, Real-Time Imaging, 4(4):263-274, 1998. In “Augmented reality simulator for training in two-dimensional echocardiography”, Computers and Biomedical Research, 33:11-22, 2000, M. Weidenbach, C. Wick, S. Pieper, K. J. Quast, T. Fox, G. Grunst, and D. A. Redel proposed to register the recorded patient-specific volumes to a generic 3D anatomical heart model, which they thus call augmented training. All the above methods present the user a non-deformable model of the anatomy. In other words, although the images may change with the transducer position/orientation in some simulators, they do not change according to the trainee's interaction with the mannequin or the virtual model, which negatively impacts the simulation realism.
Deformable Interactive Ultrasound Simulation
Compared to non deformable solutions, deformable, interactive ultrasound simulation may generate a better sense of simulator realism and consequent trainee immersion. In “B-mode ultrasound image simulation in deformable 3-D medium”, IEEE Trans Medical Imaging, 28(11):1657-69, November 2009, O. Goksel and S. E. Salcudean introduced the first interpolative simulator that allows for deformation by using a fast mapping technique for image pixels to be simulated, from the deformed simulation tissue coordinates to a nominal recorded volumetric coordinate frame. This method combines an input (reconstructed) 3D volume and interactive, volumetric tissue deformation models such as the finite element method and has been further applied to prostate brachytherapy simulation as published in Orcun Goksel, Kirill Sapchuk, and Septimiu E. Salcudean, “Haptic simulator for prostate brachytherapy with simulated needle and probe interaction”, IEEE Trans Haptics, 4(3):188-198, May 2011. It has also been applied to a transrectal ultrasound training application with fluoroscopy imaging as published in Orcun Goksel, Kirill Sapchuk, William James Morris, and Septimiu E. Salcudean, “Prostate brachytherapy training with simulated ultrasound and fluoroscopy images”, IEEE Trans Biomedical Engineering, 60(4):n2002-12, April 2013.
However, the latter methods by Goksel et al. only used a virtual patient model for simplicity. In a passive haptic VR/AR medical simulator, a physical anatomy model (mannequin) is further used with medical tools or instruments to simulate the medical procedure training as realistically as possible. For example in a state of the art embryo transfer IVF medical procedure as represented by FIG. 1, tools such as an ultrasound probe 130, a speculum 120, and an embryo transfer catheter 112 may be used. As described for instance by http://www.advancedfertilitycom/ivf-embryo-transfer-catheter.htm, a stiffer outer sheath (catheter guide) is first used to guide the soft and flexible inner embryo transfer catheter 112 through the cervical canal to the proper location in the uterine cavity 100. Once at the right location, the transfer catheter 112 is loaded with the embryos 115 which are propulsed with the embryo transfer syringe 110 into the uterine cavity 100. The whole procedure requires very careful gesture as any misposition or wound to the uterine may result in the failure of the IVF, which is a very costly and time-consuming procedure for the patients. The IVF embryo transfer procedure is conducted under ultrasound imaging supervision, by means of an abdominal ultrasound probe 130. The user manipulates the abdominal ultrasound transducer 130 by pressing it on the belly skin and positioning and orienting it to optimize the ultrasound imaging capture. An ultrasound gel or jelly is used to facilitate the ultrasound propagation as well as the manipulation of the ultrasound transducer in contact with the patient skin. In general, a moderately full bladder 140 is advisable for better ultrasound imaging quality, which will more or less deform with the probe compression on the above patient skin. The speculum tool 120 is usually made of metal and thus acts as a shield to the ultrasound waves, which also results in specific artefacts into the ultrasound images.
As known to those skilled in the art, the simulation of such a complex ultrasound procedure may raise specific issues to ensure a realistic virtual-physical dualism, e.g. the image should appear only when the ultrasound probe physically touches the anatomy model also emanating only from the curved surface of contact. Deformation of the ultrasound images and its components such as the skin (high compression), the bladder (moderate compression) and the uterine, the speculum and the catheter (no compression) needs to be synchronized with the user manipulation of the ultrasound probe, so the latter interaction remains as realistic as possible.
In “Patient-Specific Interactive Ultrasound Image Simulation with Soft-Tissue Deformation”, PhD thesis, University of California Los Angeles, 2013, K. Petrinec proposed to adapt a commercially available ultrasound simulator (www.sonosim.com) to further simulate the deformation of soft tissues when in contact with the ultrasound probe, based on the Goksel method. In this simulator, only 3-DOF motion trackers are used to track the ultrasound probe orientation—that is, the probe is assumed to be positioned at a given 3D static position over a mannequin, and the system does not track its translational movement.
More recently, as described in US20150154890, Sonosim further introduced a 5-DOF tracking solution for this simulator by adding a translation sensor as a patch that can be applied over a live body or an anatomy model, in association with an electronic tag, so that the probe translation over a curved surface may be further tracked. From a passive haptic perspective, one major drawback of this solution is the lack of realism when manipulating the probe compared to a real ultrasound procedure, as no ultrasound gel can be used with this probe according to its instructions of use (http://sonosim.com/support/). In particular, friction of the rigid plastic virtual probe tip over the anatomy model surface significantly increases when the user has to push the probe to compress the skin and get a better ultrasound image. This friction is much lower in a real procedure where ultrasound gel or jelly is used to improve the ultrasound capture, but in general, it is not desirable to use any liquid or gel in a training setup due to the overhead required to clean it.
There is therefore a need for better ultrasound simulation devices, methods and systems that facilitate a diversity of ultrasound training procedures without requiring an expensive hardware setup and cumbersome registration and calibration procedures.